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ROBUST AND LOW COST OPTICAL SYSTEM FOR 
SENSING STRESS, EMOTION AND DECEPTION IN HUMAN SUBJECTS 

BACKGROUND OF THE INVENTION 
[0001] The present invention relates generally to biometric techniques for 
sensing stress and emotion in human subjects and, more particularly, to non- 
invasive, opticar systems and methods for sensing stress and emotion. The 
underlying principle on which the present invention is based is that internal 
emotional and physiological states of a human subject are correlated with transient 
and vibratory surface motions of the skin, and that these motions provide insight 
into the subject's physiological and psychological state. Moreover, this insight can 
enable a reliable assessment of the affect and credibility of the subject through the 
application of algorithrtis based on established databases. 

[0002] More specifically, the relationships between emotions and a subject's 
facial expressions and movements have been well documented by Paul Ekman and 
others. The Facial Action Coding System (FACS), by Paul Ekman, Wallace V. 
Friesen and Joseph C. Hager, originally published in 1978 by Paul Ekman, 
University of Calif omia at San Francisco, San Francisco, California, includes an 
ACS Manual and other materials defining the emotion-expression relationships in 
detail. Dr. Ekman has authored or edited numerous other published papers and 
books in the same subject matter. See, for example, *What The Face Reveals: 
Basic and Applied Studies of Spontaneous Expression Using The Facial Action 
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Coding System (FACS)," Ekman, P. & Rosenberg, E.L. (editors), Oxford University 
Press, New York, 1997. 

[0003] The best known prior art teclinique for making such measurements of 
facial features and movements was based on the use of laser Doppler vibrometry 
(LDV) and relatively primitive electro-mechanical sensors. LDV is a technique for 
detecting surface vibration without physical contact and is widely used in 
engineering and manufacturing applications. 

[0004] As useful as LDV has proven to be for making measurements of facial 
movements, the method has significant drawbacks. A laser beam must be pointed 
at a specific test point on the subject's skin that is believed to be indicative of an 
emotional state. The vibrations or transient motions induce frequency modulation on 
light that is reflected back to the instrument, and well-known heterodyne laser 
detection nriethods are used to recover the mechanical signal for display and 
processing. The beam must be maintained on the test point to make the 
measurement, and this requirement presents some practical problems, in general 
requiring that the subject's movements be constrained to some degree during the 
test. 

[0005] A more serious limitation is that only a single point can be examined at 
a time with conventional LDV hardware. While it is known that monitoring many 
points on the subject's skin simultaneously provides significant information 
concerning the subject's affect, a separate LDV instrument would be required to 
monitor the response at each point to capture all of the spontaneous facial 
movements needed to detect a particular emotional condition. The pioneering work 
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of Ekman and Friesen on micro-expressions showed that many facial muscle 
groups are involved Jn the involuntary transient expression of emotion. For a brief 
time of about 300 ms (milliseconds) or so, the face of a subject responds 
involuntarily to emotion-evoking stimuli such as questions, comments, or scenes, 
and these fleeting expressions can be detected and interpreted. The Facial Action 
Coding System developed by Ekman et al. maps specific emotions to coordinated 
motions in more that twenty facial muscle groups. Hence, being' unable to monitor 
all or most of these muscle groups at the same time is a serious limitation for LDV 
instruments. 

[0006] Lastly, signal processing for the received signal in an LDV instrument 
is somewhat problematic, requiring algorithms to remove subject motion artifacts. 
The LDV is useful for making high fidelity measurement of skin surface movements 
for cardiology and other applications, but it would be highly desirable if the motions 
of an entire face could be sensed at an instant, so that correlated responses among 
muscle groups could be detected and processed. The present invention is directed 
to this end. 

SUMMARY OF THE INVENTION 
[0007] The present invention resides in a metliod and apparatus for sensing 
emotional and physical conditions of a human subject by evaluating movements in 
selected areas of the subject's face. Briefly, and in general terms, the method of the 
invention comprises the steps of generating an image of substantially all of the face 
of a human subject; processing the image to identify movements in selected critical 
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areas of the face; comparing the identified movements in the selected critical areas 
with a database that associates movements in selected critical areas with specific 
emotional and physical conditions; and generating a report of the emotional and 
physical condition of the subject. 

[0008] More specifically, the processing step comprises inputting a two- 
dimensional frame of the image; scanning the image to locate the subject's face and 
determine it's relative position and extent; scanning the facial part of the image to 
detect the selected critical areas; repeating the preceding steps for a sequence of 
image frames; recording frame-to-frame changes in critical areas of interest; and 
recording frame-to-frame changes in critical area positions, for purposes of tracking 
the positions while permitting limited movement of the subject. 
[0009] In apparatus temns, the invention comprises an optical imaging device, 
for generating an image of substantially all of the face of a human subject; an image, 
processing module, for processing the image to identify movements in selected 
critical areas of the face; a database that associates groups of facial movements 
with specific emotional and physical conditions of the subject; a database analysis 
module, for comparing the identified movements in the selected critical areas with 
the database; and a report generator, for generating a report of the emotional and 
physical condition of the subject. 

[0010] It will be appreciated from the foregoing brief summary that the 
present invention represents a significant advance in the field of human emotion 
sensing apparatus and methods. In particular, the present invention images 
substantially the entire facial image of the subject, and senses and tracks multiple 
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critical areas of the image simultaneously, comparing the results with a database to 
obtain an assessment of the subject's emotional and physical condition. Other 
aspects and advantages of the invention will become apparent from the following 
more detailed description of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0011] Figure 1 is block diagram showing the principal components of the 
present invention. 

[0012] Figure 2 is a flowchart depicting the principal functions performed by 
an image processing module in the present invention. 

[0013] Figure 3 is a flowchart depicting the principal functions performed by a 
database analysis module in the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
[0014] As shown in the drawings for purposes of illustration, the present 
invention is concerned with an optical technique for detecting involuntary 
movements of the face of a human subject, and using the detected movements to 
report various emotional conditions, such as stress and deception, experienced by 
the subject. Prior to the present invention, detection of movement in a facial feature 
of interest has been accomplished using a laser beam and a technique known as 
laser Doppler vibrometry (LDV). The LDV approach has a number of significant 
disadvantages, which have been discussed above. 
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[0015] in accordance with the invention, a human subject's entire face is 
rapidly scanned to detect movements in critical areas that are known to be affected 
involuntarily when the subject is exposed to various emotion-provoking stimuli, and 
the detected responses are compared with a database that associates the 
responses with specific emotions or physiological conditions. As shown In Figure 1, 
a subject, indicated diagrammatically by a face 10, is imaged by a speckle detection 
and tracking sensor 12. The term "speckle" is derived from "laser speckle," a 
sparkling granular pattern that is observed when .an . object diffusely reflects 
coincident laser light. The laser speckle pattern has been used to make surface 
measurements of objects, with techniques known as speckle metrology or speckle 
interferometry. In the present context, use of the term "speckle" is not intended to 
limit the invention to the use of lasers to illuminate the subject. On the contrary, the 
invention is intended to operate using available light or, as will be further discussed, 
a narrow-band source outside the visible spectrum. . . 
[0016] The sensor 12 may be any two-dimensional full-frame digital camera 
device, using, for example, CCD (charge-coupled device) technology or CMOS 
(complementary metal-oxide semiconductor) imaging devices. If laser illumination is 
used, the sensor 12 may use electronic speckle pattem interferometry (ESPI), such 
as the ESPI sensors made by Steinbishler Optotechnik GmbH, as described on the 
Web site of the Syncretek company, at www.svncretek.com . 
[0017] Image data produced by the sensor 12 are processed in an image 
processing module 14 to detect and track "speckle spots" on the subject's face, as 
described more fully with reference to Figure 2. The processed image may be 
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supplied to a display 16 for use by an operator of the apparatus. Data concerning 
the identified spots of interest on the subject's face are transferred to a database 
analysis module 18, which compares the identified spots with a database of known 
associations between facial movements and emotional and physiological conditions. 
From this comparison, the database analysis module 18 generates an assessment 
or report 20. The assessment may be merged with the display data fed to the 
display 16, to alert the operator to critical conditions or conclusions concerning the 
face of the subject 10. 

[0018] Processing the captured image of the subject 10 can take various 
forms. The basic processing steps performed in the image processing module 14 
are shown in Figure 2. After a new frame of the image has been input to the 
processing module 14, as indicated by block 24, the next step is to scan the image 
to locate the face position'and its extent in the image, as indicated in block 26. The 
face outline and position are located with reference to its known standard features, 
such as ears, eyes, nose, mouth arid chin. Facial feature extraction is known in the 
art of biometrics, and various techniques for identifying and Ideating the principal 
facial features have been the subject of research and publication. For example, the 
following patents disclose such techniques: Patent No. US 6,600,830 B1, issued 
July 29, 2003 to Chun-Hung Lin and Ja-Ling Wu, entitled "Method and System of 
Automatically Extracting Facial Features," and Patent No. US 6,526,161 B1, issued 
February 25, 2003 to Yong Yan, entitled "System and Method for Biometrics-Based 
Facial Feature Extraction." To the extent that these two patents are deemed 
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necessary to a complete disclosure of the present invention, they are hereby 
incorporated by reference into this description. 

[0019] Once the face and its principal features have been located within the 
two-dimensional image, the next step is to detect and locate critical muscle spots 
that are known to be subject to vibration or transient movement when the subject is 
exposed to emotion-evoking stimuli. The positions of these critical muscle spots 
with respect to the principal facial features are known in advance, at least 
approximately, from the works of Ekman and others, and particularly from Ekman's 
Facial Action Coding System. The locations of the muscle spots or "speckle spots" 
can be more precisely determined using any of at least three algorithmic search 
methods. ' 

[0020] One method for locating the critical spots is 2-D (two-dimensional) 
image motion sensing, i.e., the detection of repetitive fluctuation of reflected light in 
the speckle spot, corresponding to facial muscle vibrational movements. This 
algorithmic approach enables detection and location acquisition by means of a 
processing algorithm using the inputted 2-D imaging pixel data, which then looks for 
local multiple-pixel reflectivity fluctuations (frame to frame), compared to non- 
vibratory areas of the adjacent facial surfaces. The frame rate must be high enough 
to sense the peaks and valleys of speckle reflectivity changes. 
[0021] Another approach is 3-D (three-dimensional) dimple motion sensing. 
Dimple motion is a repetitive fluctuation of speckle spots orthogonal to facial skin, 
equivalent to dimples that can sometimes be visually observed. Orthogonal, in this 
context, means in the same direction as the camera focal axis. Dimpling must be 
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sensed as a change in distance from the camera or sensor 12. The dimpling 
movement of the speckle spot is driven by vibratory local facial muscles. This 
algorithmic approach can be achieved using range measurement 3-D, full frame 
camera methods. The range resolution must be compatible with expected changes 
in dimple/speckle movements, and should be no more than approximately 0.5 mm 
or slightly larger. 

[0022] Another approach is to use double sampled (pulse frame differenced) 
imaging for 2-D or 3-D motion sensing. This method utilizes an actively illuminated, 
pulsed source at a near infrared (IR) wavelength, which provides cancellation of 
background illumination (sunlight, shadowing) by subtracting the 2-D image at the 
end of the pulse period, from the 2-D image at the beginning of the pulse period. 
The principal advantage, of the method is a high contrast ratio in the resulting image, 
since slow variations in both high brightness (direct sunlight), and shadowed 
environments (outdoors or indoor) will be eliminated. This makes a subsequent 
motion sensing step much more accurate. 

[0023] As indicated in block 30, image processing includes recording frame- 
to-frame changes in the size. and axial distance of sport of interest. As indicated 
above, such changes are used in various approaches to detect the presence and 
locations of, the spots initially, as well as to detect changes in the spots in terms of 
their extent and axial distance, as measured over a selected time inten/al. As 
indicated in block 32, there is also a requirement to track frame-to-frame positions 
of spots in order to compensate for movement of the subject or the subject's face. In 
general, tracking and temporal recording of the speckle spots is effected by 
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measuring the continuing temporal occurrence and magnitude intensity changes of 
the spots. This is the desired data that will be both stored and temporally marked to 
correlate to other events (e.g., questions from the examiner) to sense the emotional 
behavior and status of the individual being examined. 

[0024] The database analysis module 18 (Figure 1) perfomns the steps 
outlined in Figure 3. As indicated in block 40, the image data provided by the image 
processing module 14 are categorized as needed for a particular application. For 
example, in the detection of deception by the subject 10, only a subset of all the 
spots detected and processed may be needed for deception analysis. The spots of 
interest are categorized in terms of their size and activity during a selected period of 
time, and then submitted to the next step of analysis, in which the selected and 
categorized spot data are compared, in block 42,^ with database parameters 
retrieved from a facial action coding system (FACS) database 44. The database 44 
contains a list of* all relevant combinations of speckle spot parameters, stored in 
association with corresponding emotions or physiological conditions. Based on this 
comparison with the database, the apparatus generates a report, as indicated in 
block 46.. In addition, selected conclusions reached as a result of the analysis are 
transmitted to .the display 16, where they are overiaid with the facial image to 
provide the operator with a rapid feedback of results, together with an indication of a 
reliability factor based on the degree to which the detected spot movements 
correlate with the database indications of an emotion, such as deception. In addition 
to this result information, the display 16 may also be overiaid with color-coded 
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indications of muscle spot activity, which an experienced operator should be able to 
interpret and take appropriate action. 

[0025] The various embodiments described have in common the utilization of 
full-frame imaging, speckle detection and tracking. The image is processed in 
several ways to provide useful infomnation. The areas on the subject's face that 
respond to stimuli can be highlighted in the display. The frequency and duration of 
the response in an area can be determined and presented with an appropriate color 
code or other legend to aid the operator. Thresholds and filters can be implemented 
using conventional digital processing techniques, to maximize the sensitivity and of 
the instrument in detecting stress and emotional state. The pattems^of excitation 
may also be compared to a database obtained through testing a representative 
population, perhaps differentiated by factors such as age, gender, ethnic origin, and 
so forth. By applying appropriate algorithms to the imjage data, . physiological 
parameters such as piilse and cardiac waveform can be obtained along with an 
analysis of the patterns of facial muscle group excitations. 

[0026] It will be appreciated from the foregoing that the present Invention 
represents a significant advance in the field of facial analysis for the purpose of 
detecting various emotions and associated physiological conditions that manifest 
themselves when a subject is exposed to emotion-provoking stimuli. In particular, 
the invention provides for the detection and analysis of multiple facial muscle 
groups simultaneously, during a brief period of involuntary activity following 
application of an external stimulus: It will also be appreciated that, although specific 
embodiments of the invention have been described in detail for purposes of 
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illustration, various modifications may be made without departing from the spirit and 
scope of the invention. Accordingly, the invention should not be limited except as by 
the appended claims. 
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